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•Paradigm for pre/post data. 

•How should we describe change? 

•Common analysis methods for comparing post to pre results. 
•What do we mean by “% change”? 

•What are we testing when we compare % changes? 

•Some simulation results 
•Conclusions 



Characteristics of Pre/Post data 


repeated-measures design 


subject 

pre 

post 

subject 

session 

meas. 

1 

36.4 

31.5 

1 

L-60 

50.3 

2 

48.8 

40.1 

1 

L-10 

48.6 

3 

25.9 

25.6 

1 

R+0 

35.7 

etc 



1 

R+3 

39.4 


1 

R+10 

46.0 

2 

L-60 

75.0 

2 

L-10 

78.5 

2 

R+0 

69.6 

2 

R+3 

73.5 

2 

R+10 

77.9 


etc 
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Characteristics of Pre/Post data (cont.) 

partially repeated-measures design 


subject 

session 

meas. 

group 

1 

L-60 

50.3 

C 

1 

L-10 

48.6 

C 

1 

R+0 

35.7 

C 

1 

R+3 

39.5 

C 

1 

R+10 

46.0 

C 

2 

L-60 

75.0 

C 

2 

L-10 

78.5 

C 

2 

R+0 

69.6 

C 

2 

R+3 

73.5 

C 

2 

R+10 

77.9 

C 

etc 

• • • 

• • • 

C 


subject 

session 

meas. 

group 

9 

L-60 

50.3 

E 

9 

L-10 

48.6 

E 

9 

R+0 

35.7 

E 

9 

R+3 

39.4 

E 

9 

R+10 

46.0 

E 

10 

L-60 

75.0 

E 

10 

L-10 

78.5 

E 

10 

R+0 

69.6 

E 

10 

R+3 

73.5 

E 

10 

R+10 

77.9 

E 

etc 

■ • • 

• a ■ 

E 
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How should we express the effect of an 
intervention? 


•mean change? 

•% change? 

•mean change in log measurements? 



How should we express the effect of an 
intervention? 

•mean change 


pre 

post 

dif 

% ch 

75 

85 

10 

13.3% 

100 

111 

11 

11.0% 


6 


How should we express the effect of an 
intervention? 

•% change 


pre 

post 

dif 

% ch 

10 

18 

8 

80% 

30 

51 

21 

70% 


How should we express the effect of an 
intervention? 

•mean change in log measurements 


pre 

post 

dif 

% ch 

10 

18 

8 

80% 

1000 

1700 

700 

70% 
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How should we express the effect of an 
intervention? 

•mean change in log measurements 


pre 

post 

dif 

% ch 

10 

18 

8 

80% 

1000 

1700 

700 

70% 


•Observed outcome measurements are > 0 

•They can range over one or more orders of magnitude 
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How should we express the effect of an 
intervention? 


•mean change? 

•% change? 

•mean change in log mesaurements? 


pre 

post 

dif 

% ch 

75 

85 

10 

13.3% 

100 

110 

10 

10.0% 

130 

140 

10 

7.7% 
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How should we express the effect of an 
intervention? 


•mean change? 

•% change? 

•mean change in log mesaurements? 


pre 

post 

dif 

% ch 

75 

85 

10 

13.3% 

100 

110 

10 

10.0% 

130 

140 

10 

7.7% 


Ans. Correct interpretation should be 
driven by clinical relevance. 
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pre and post- bedrest data 


sub 

pre 

post 

dif 

1 

88.7 

88.2 

-0.5 

2 

85.1 

102.1 

16.9 

3 

106.3 

98.6 

-7.6 

4 

115.6 

96.2 

-19.4 

5 

62.6 

77.3 

14.8 

6 

85.4 

82.5 

-2.9 

7 

93.1 

97.8 

4.7 

8 

87.1 

36 

-51.1 

9 

80.7 

64.6 

-16.1 

10 

138.6 

111.5 

-27.1 
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Typical approaches to inference on 
the effect of bedrest: 


subject 

pre 

post 

diff 

1 

36.4 

31.5 

-4.9 

2 

48.8 

40.1 

-8.7 

3 

25.9 

25.6 

-0.3 


etc 


RPM ANOVA with phase (pre/post) as a factor 

t-test of H 0 : mean diff = 0 

t-test of H 0 : pet change of means = 0 

t-test of H 0 : mean pet change = 0 
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Typical approaches to inference on 
the effect of bedrest: 


subject 

pre 

post 

diff 

1 

36.4 

31.5 

-4.9 

2 

48.8 

40.1 

-8.7 

3 

25.9 

25.6 

-0.3 


etc 


RPM ANOVA with phase (pre/post) as a factor (ju post = ju pre 

t-test of H 0 : mean diff = 0 (// posf = ju pre ) 

t-test of H 0 : pet change of means = 0 {ju post = ju pre ) 

t-test of H 0 : mean log pre = mean log post (£ post = Q re ) 
t-test of H 0 : mean observed pet change = 0 (???) 


AN OVA 


Number of obs = 20 

Root MSE = 14.526 


R-squared = 0.7828 

Adj R-squared = 0.5414 


Source 


Partial SS 

df 

MS 

F 

Prob > F 

Model 

1 

1 

1 

6843 . 84439 

10 

684.384439 

3.24 

0 . 0455 

isub 

1 

1 

6453 .49865 

9 

717 . 055406 

3.40 

0 . 0414 

post 

1 

390 . 345734 

1 

390.345734 

1.85 

0.2069 

Residual 

1 

1 

1899.05337 

9 

211 . 005931 











Total 


8742 .89776 


19 460.152514 


t-test on differences 


Variable | 


Obs 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. 

Interval ] 

1 

dif | 


10 

-8 . 835675 

6.496244 

20.54293 

-23 . 5312 

5 . 85985 

mean = 

mean (dd) 




t = 

-1.3601 

Ho: mean = 

0 




degrees 

of freedom 

9 

Ha: mean 

< 0 


Ha: mean != 

0 

Ha: mean > 0 

Pr (T < t) 

= 

0 . 1034 

Pr ( | T | 

> 1 1 1 ) = 

0.2069 

Pr (T > 

t) = 0.896 
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Percent change in means 


Variable | 

Obs 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. 

Interval ] 

1 

post | 
pre | 

10 

10 

85.4812 
94 .31687 

6.958522 

6.662214 

22.00478 

21.06777 

69.73992 

79.24589 

101.2225 

109.3878 

1 

dif f | 

10 

-8 . 835675 

6.496244 

20.54293 

-5 . 85985 

23 . 5312 


Testing r u 



= 100 


y post y pre 



0 is the same as testing ju post = ju pre . s 
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t-test on differences of logs 


One-sample t test 


Variable | 


Obs 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. Interval] 

1 

log AES | 


10 

-.1180426 

.097688 

.3089166 

-.3390283 .102943 

mean = 

mean ( z ) 




t = -1.2084 

Ho : mean = 

0 




degrees 

of freedom = 9 

Ha: mean 

< 0 


Ha : mean ! = 

= 0 

Ha: mean > 0 

Pr (T < t) 

= 

0.1288 

Pr ( 

|T| > | t | ) = 

= 0.2577 

Pr (T > t) = 0 . 8712 
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t-test on observed percent changes 


One-sample t test 


Variable | 


Obs 

Mean Std. Err. 

Std. Dev. 

[95% Conf. Interval] 

1 

%ch AES | 


10 

-.0775129 .0741236 

.2343994 

-.2451921 .0901664 

mean = 

mean (pch) 



t = -1 . 0457 

Ho : mean = 

0 



degrees 

of freedom = 9 

Ha: mean 

< 0 

Ha : mean ! = 

= 0 

Ha: mean > 0 

Pr (T < t) 

= 

0 .1615 

Pr(|T| > | t | ) = 

= 0.3230 

Pr (T > t) = 0.8385 
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t-test on observed percent changes 


One-sample t test 


Variable | 


Obs 

Mean Std. Err. 

Std. Dev. 

[95% Conf. Interval] 

1 

%ch AES | 


10 

-.0775129 .0741236 

.2343994 

-.2451921 .0901664 

mean = 

mean (pch) 



t = -1 . 0457 

Ho : mean = 

0 



degrees 

of freedom = 9 

Ha: mean 

< 0 

Ha : mean ! = 

= 0 

Ha: mean > 0 

Pr (T < t) 

= 

0 .1615 

Pr(|T| > | t | ) = 

= 0.3230 

Pr (T > t) = 0.8385 


What are we testing when we do this? 
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Objective: compare pre/post means 


subject 

pre 

post 

diff 

1 

36.4 

31.5 

-4.9 

2 

48.8 

40.1 

-8.7 

3 

25.9 

25.6 

-0.3 


etc 


v' RPM ANOVA with phase (pre/post) as a factor 
v' t-test of H 0 : mean diff = 0 
S t-test of H 0 : pet change of means = 0 

s t-test of H 0 : mean log pre = mean log post 

? t-test of H 0 : mean observed pet change = 0 
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“Bone density is decreased by 6% after 
ISS missions.” 
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“Bone density is decreased by 6% after 
ISS missions.” 


% change in means = -6% ? 
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“Bone density is decreased by 6% after 
ISS missions.” 


% change in means = -6% ? 


mean % change = -6% ? 
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“Bone density is decreased by 6% after 
ISS missions.” 


% change in means = -6% ? 


mean % change = -6% ? 



What’s the difference? 




pet change in means 


estimated by 
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pet change in means 


estimated by 


1 00 x — post ^ pre 




pre 


100x 


y post y pre 


y 


pre 


f 


mean pet change Mpc = E 


lOOx ^ i,post ^^P re 




V 


Mi 


pre J 


estimated by 


-Z^lOOx 

n i 



?? 


(average observed pet change) 
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An Example 

subject pre post diff % ch 

1 30 25 -5 -16.7 

2 40 36 -4 -10.0 

3 25 19 -6 -24.0 

4 53 49 -4 -7.5 

ave 37 32.25 -4.75 -14.6 


estimated % change in means = 100 x (-4.75/37 ) = 


estimated mean observed % e = -14.6 
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r 


mean obs pet change W -E 


IQQ x y impost y i,pre 


\ 


V 


yi 


pre J 


— y<i ioox 


1 

n i 


y 


i,pre 


for “large” n 
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r 


mean obs pet change W -E 


IQQ x y impost y i,pre 


\ 


V 


yi 


pre J 


— y<i ioox 


1 

n i 


y 


i,pre 


for “large” n 


If it exists, ¥ is a population characteristic, 
but what does it mean? 
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r 


mean obs pet change W -E 


lOOx 


y impost yi,pre 


\ 


V 


yi 


pre J 


— y<i ioox 


i 

n i 


y 


i,pre 


for “large” n 


If it exists, ¥ is a population characteristic, 
but what does it mean? 


( 


Is W the actual mean pet change Mpc = E 


1 QQ x r*j,post Mj,pre 


\ 


V 


Mi. 


pre J 
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r 


mean obs pet change W -E 


IQQ x y impost y i,pre 


\ 


V 


yi 


pre J 


— y<i ioox 


1 

n i 


y. 


i, pre 


for “large” n 


If it exists, Y is a population characteristic, 
but what does it mean? 


( 


Is Y the actual mean pet change Mpc = E 


lOOx 


Mi 


v 


post 

Mi 


M 


\ 


i, pre 


pre 


J 


? 

■ 


If not, how well does Y describe the effect of 
spaceflight/bedrest over a population of subjects? 
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Simulation 1 


•10,000 “experiments” 

•N = 20 simulated paired pre-post measurements of 
ankle extensor strength (AES) per experiment 
•mean effect of “bedrest” : A = /u post - ju pre 

•t-test #1 - test H 0 : A = 0. 


•t-test #2 - test H' 0 : 'F = 0. 



actual AES data (N = 24, pre/post) 


AES 

AES 


LO 

o 


CO 

c 

CD 

O 


LO 

o 

o 


Variable 

(pre) 

(post 


■ + ■ 


1 

40 

i i i i 

60 80 100 120 
ankle extensor strength 

l 

140 


Obs 

Mean 

Std. Dev. 

Min 

Max 

24 

103.0 

23.0 

67 . 8 

152 . 1 

24 

82.8 

22 . 9 

42.2 

128 . 1 
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Simulation 1 (cont.) 


■distribution of pre-means (between subjects) 

■distribution of post-means (between subjects) 

■distribution around subject-specific means (within subjects) 
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Simulation 1 (cont.) 


■distribution of pre-means (between subjects) 

■distribution of post-means (between subjects) 

■distribution around subject-specific means (within subjects) 

•constant SD 
•constant CV (SD/mean) 

•other ? 
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X 


36 



X 


37 



post 


AES strength 


actual data 



40 60 80 100 120 140 160 

pre 
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post 

100 120 140 160 


AES strength 

actual data 
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Simulation 1 (cont.) 


10,000 “experiments” 


•N = 20 simulated paired pre-post measurements of 
ankle extensor strength ( AES) per experiment 


•mean effect of “bedrest” : A = {i post - ju pre = 0 


t-test #1 - test H 0 : A = 0. A = E{y ipost - y i pre ) 


,pre 


f 


•t-test #2 - test H' 0 : W = 0. *¥ = E 


lOOx 


y impost yi,pre 


V 


7 


i,pre 


\ 


J 
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Simulation 1 (cont.) 


•10,000 “experiments” 


•N = 20 simulated paired pre-post measurements of 
ankle extensor strength ( AES) per experiment 


•mean effect of “bedrest” : A = {i post - ju pre = 0 


•t-test #1 - test H 0 : A = 0. A = E[y t 


post 


yi 


pre 


) 


•t-test #2 - test H' 0 : 'F = 0. 


f 


'F = E 


lOOx 


Vi, post si, pre 


\ 


y. 


i,pre 


\ 


J 


Output: 

•10,000 “t”-values and p-values for t-test #1 
•10,000 “t”-values and p-values for t-test #2 



Expected Results 


If H 0 (A = 0) is true: 

t-values for test 1 should be distributed as t(19) 
p-values for test 1 should be distributed as U(0,1) 

If H' 0 : (¥ = 0) is true: 

t-values for test 2 should be distributed as t(19) 
p-values for test 2 should be distributed as U(0,1) 
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Test 1 H 0 :|g= 0; N = 20 

- 


CO _ 



-5 -4 -3 -2 -1 0 1 2 3 4 5 

t-value 
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Test 2 HoH =0; N = 20 
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# P < .05 = 499 


Test 1 H„: □ = 0; N = 20 


in 


c/) 

c 

CD 

O 


LO 


I 

0 


~r~ 

.4 


p-value 


~r~ 

.6 


.8 
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# P < .05 = 558 


Test 2 HM =0; N = 20 


LO 


C/) 

C 

CD 

O 


LO 


I 

0 


i r~ 

.4 .6 

p-value 


~r~ 

.8 
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Simulation 2 


•10,000 “experiments” 


•N = 60 simulated paired pre-post measurements of 
ankle extensor strength ( AES) per experiment 


•mean effect of “bedrest” : A = {i post - ju pre = 0 


•t-test #1 - test H 0 : A = 0. A = E[y t 


post 


yi 


pre 


) 


•t-test #2 - test H' 0 : 'F = 0. 


f 


'F = E 


lOOx 


Vi, post si, pre 


\ 


y. 


t,pre 


\ 


J 


Output: 

•10,000 “t”-values and p-values for t-test #1 
•10,000 “t”-values and p-values for t-test #2 



- 4 — * 
'(/) 
c 
CD 

Q 


C\J _ 


<=> “4 


-5 


~r 

-4 


Test 1 H 0 : □ = 0; N 


-3 - 2-10 1 

t-value 


60 



~r 

5 
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Test 2 Holl = 0; N = 60 


_ 



-5 -4 -3 -2 -1 0 1 2 3 4 5 

t-value 
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# P < .05 =492 


Test 1 H„: □ = 0; N = 60 


in 


c 

CD 

O 


LO 


I 

0 


~r~ 

.4 


p-value 


~r~ 

.6 


~r~ 

.8 
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# P < .05 = 985 



p-value 


60 


.8 1 
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What’s going on ?? 
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What’s going on 


Ans: *¥ * 0 
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Estimating ¥ 


Simulation of pre, post AES values with A = ju post - ju pre = 0 


f 


V F = E 


lOOx 


si, post si, pre 


\ 


y 


i,pre 


\ 


J 


N = 10 3 , 10 4 , 10 5 , and 10 6 
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Estimating ¥ 

Simulation of pre, post AES values with A = ju post - ju pre = 0 


f 


V F = E 


lOOx 


si, post si, pre 


\ 


y 


i,pre 


\ 


J 


N 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. 

Interval ] 

1000 

1 . 950 

. 639 

20.2 

0.696 

3.204 

10000 

2.028 

.207 

20.7 

1 . 623 

2 . 435 

100000 

2.079 

.066 

20.9 

1 . 949 

2.209 

1000000 

2.048 

.021 

20.9 

2 . 007 

2 . 089 


55 


Estimating ¥ 


Simulation of pre, post AES values with A = ju post - ju pre = 0 


f 


V F = E 


lOOx 


si, post si, pre 


\ 


y 


i,pre 


\ 


J 


N 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. 

Interval ] 

1000 

1 . 950 

. 639 

20.2 

0.696 

3.204 

10000 

2.028 

.207 

20.7 

1 . 623 

2 . 435 

100000 

2.079 

.066 

20.9 

1 . 949 

2.209 

1000000 

2.048 

. 021 

20.9 

2 . 007 

2 . 089 


A = 0, |LXp C = 0, but = 2.05 % ! 
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Estimating ¥ 


Simulation of pre, post AES values with A = ju post - ju pre = 0 


f 


V F = E 


lOOx 


si, post si, pre 


\ 


y 


i,pre 


\ 


J 


N 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. 

Interval ] 

1000 

1 . 950 

. 639 

20.2 

0.696 

3.204 

10000 

2.028 

.207 

20.7 

1 . 623 

2 . 435 

100000 

2.079 

.066 

20.9 

1 . 949 

2.209 

1000000 

2.048 

. 021 

20.9 

2 . 007 

2 . 089 


A = 0, p PC = 0, but W = 2.05 % ! 

Does this mean that sham bedrest causes AES to 
increase by 2.05% ??? 




¥ depends on the with in-subject CV (sd/mean) 


CV 

Y (%) 

0.01 

0.01 

0.05 

0.23 

0.10 

1.04 

0.12 

1.55 

0.14 

2.05 

0.16 

2.70 

0.18 

3.53 

0.20 

4.53 
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A single pre value is not a stable baseline ... 



# 63 


pre 


post 


next 


What if A ^ 0 ? 
(mean post * mean pre) 


pre mean = 91.75 
post mean = pre mean + A 


n 

A 

pow(d) 

pow(pct ch) % ch mean 


20 

-11.5 

0.82 

0.74 

-24.5 

-11.1 

30 

-9.2 

0.80 

0.65 

-10.0 

-8.3 

40 

-7.8 

0.78 

0.59 

-8.5 

-6.5 

60 

-6.4 

0.79 

0.56 

-7.0 

-5.5 
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What about data varying over several orders of 
magnitude? 

Should we analyze logs or percent changes? 



Density 


t-test on differences 


CM 


LO 


LO 


~T~ 

0 


1 1 

.4 .6 

p-values for test on differences 


.8 
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Density 


t-test on pet change 


LT) 


LO - 


0 .2 .4 .6 .8 1 

p-values for pet change 
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Density 


t-test on differences of logs 


LO 


LO 


I 

0 


1 1 

.4 .6 

p-values for test on logs 


.8 
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Comparing experimental groups of subjects 


sub 

pre 

post diff 

group 

i 

36.4 

31.5 -4.9 

c 

2 

48.8 

40.1 -8.7 

c 

3 

25.9 

19.6 -6.3 

c 

a a a 

a a a 

a a a a a a 

c 

10 

40.5 

38.2 -2.3 

c 

1 

36.4 

31.5 -4.9 

E 

2 

48.8 

46.1 -2.7 

E 

3 

25.9 

25.6 -0.3 

E 

a a a 

15 

a a a 

37.7 

a a a a a a 

37.0 -0.7 

t 

E 


^2 
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Comparing experimental groups of subjects 

Equal CV’s 

Bias in average observed % change 
approximately cancels so result of inference 
using % change as data and comparing means 
are about the same as t-test on differences. 
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Comparing experimental groups of subjects 

Equal CV’s 

Bias in average observed % change 
approximately cancels so result of inference 
using % change as data and comparing means 
are about the same as t-test on differences. 


But what does it mean to report average 
values of % change for each group? 
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Comparing experimental groups of subjects 


Unequal CV’s 


Biases in averages of observed % change 
do not cancel and thus inference itself is 
biased (size of test not 0.05) 
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Conclusions 

If outcome measure >0 and does not vary over several orders 
of magnitude, analyze original data. 

You can estimate or make inference on the % change in the 

mean response from the above 

analysis. 
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Conclusions (cont.) 

Within an experimental group , there is a positive bias when 

analyzing individual percent changes as data. 

• Average observed percent change does not estimate 
either the population mean percent change nor the 
percent change in the population means. 

• Type I error rate > a (e.g. 0.05) when no actual effect 

• Reduced power when there really is an effect 

• Effect of bias on inference becomes more evident as 
sample size increases. 
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Conclusions (cont.) 


When comparing experimental groups , if CV’s are similar, 
much of the bias cancels, thus the results of inference on 
mean % change is similar to results of inference on 
differences or ANOVA. 

However difficult to characterize each group in terms of 
average observed % change because of the individual 
biases. 

If CV’s differ, inference on % change is biased (a-level not 
0.05). 
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Conclusions (cont.) 


If outcome measure >0 and does vary over several 
orders of magnitude, analyze logs of original data. 

• stabilizes variance 

• ANOVA, t-tests perform as advertised 

• ANOVA, t-tests perform poorly on original data 

• analysis of pet change is also biased as before 
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Next time: 


•Improving estimation of mean percent change (p PC ) 

•Using regression models that use pre-data to help explain 
post-data. 

•Regression to the mean and how it can lead to misleading 
conclusions. 

•Instrumental variables to correct bias in regression models 
with random errors in predictors. 
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